Method for Segmenting and Tracking Content in Videos Using Low-Dimensional Subspaces and Sparse Vectors

ABSTRACT

A method segments and tracks content in a video stream including sets of one or more images by first determining measured data from each set of one or more images. An adaptive step-size parameter and a low-dimensional subspace characterizing motion of the content the measured data are initialized. A sparse vector representing a sparse component that characterizes the motion of the content different from the motion of the content characterized by the low-dimensional subspace is determined. A change in the low-dimensional subspace for the measured data is determined using a proximal point iteration and the parameter, which is updated according to the change. A low-rank subspace matrix representing the low-dimensional subspace is updated according to the change and the parameter. Then, the low-rank matrix representing the low-dimensional subspace and the sparse vector are outputted.

FIELD OF THE INVENTION

This invention relates generally to computer vision, and moreparticularly to segmenting and tracking content in videos.

BACKGROUND OF THE INVENTION

The problem of segmenting and tracking low-dimensional subspacesembedded in high dimensional data arises in many applications such asbackground segmentation, anomaly detection, motion segmentation, andtarget localization. For example, a scene acquired by a stationary ormoving camera can be partitioned into a low-rank component spanning asubspace that characterizes a relatively stationary background in thescene, and a sparse component corresponding to moving objects in thevideo scene, usually in the foreground.

The problem is to identify, at every time step t, e.g., each image in asequence of images, an r-dimensional subspace

_(t) in

^(n) with r<<n that is spanned by columns of a rank-r matrix U_(t) ∈

^(n×r) from incomplete and noisy measurements

b _(t)=Ω_(t)(U _(t)α_(t) +s _(t)),   (1)

where Ω_(t) is a selection operator that specifies subset of the sets ofone or more images at time t, α_(t) ∈

^(r) are coefficients specifying linear combination of columns of thesubspace U_(t), and s_(t) ∈

^(n) is a vector of sparse outliers.

When the subspace

_(t) is stationary, the subscript t is omitted from U_(t) and theproblem reduces to matrix completion or principal component analysis(PCA) where the task is to separate a matrix B ∈ E

^(n×m) into a low-rank component UA, and a sparse component S usingincomplete measurements

B _(Ω)=Ω(UA+S).

The columns of the matrices A and S are respectively the vectors α_(t)and s_(t) stacked horizontally for all t ∈ {1 . . . m}, and theselection operator Ω specifies the measured data in the matrix B.

Conventional methods for low-dimensional subspace identification firstorganize the measured data into a matrix and then determine basisvectors that span the subspace using a variety of techniques, e.g.,low-rank matrix factorization.

Extensions of those methods factor the matrix into a low-rank componentcorresponding to the subspace, and a sparse component that representsnoise.

However, when the dimensionality of the data becomes large, as is thecase of a video, latency becomes a problem. Hence, it is necessary toprovide a method that can segment and track the low-dimensional subspaceas the data are acquired or processed in real-time, even when the dataare incomplete and corrupted by sparse noise. Another problem is thatthe low-dimensional subspace (background) can vary over time, in whichcase the subspace cannot be represented by a low rank matrix when alldata are grouped into one matrix. For example, the background in outdoorscene can vary in illumination during the day. Similarly, erstwhilemoving objects can be added or removed from the background, where theare stationary, in surveillance videos.

One prior art method, U.S. Pat. No. 7,463,754, “Adaptive probabilisticvisual tracking with incremental subspace update,” describes a methodfor adaptive probabilistic tracking of an object as represented in amotion video. The method identifies an eigenbasis that represents theobject being tracked. A maximum a posteriori estimate of the objectlocation is determined using the current estimate of the eigenbasis. Theeigenbasis is then updated to account for changes in the appearance ofthe target object.

Another prior art, US 20030108220, “Robust, on-line, view-basedappearance models for visual motion analysis and visual tracking,”describes learning an appearance model that includes both a stable modelcomponent, learned over a long time course, and a transient component,learned over a relatively short time course. The model parameters areadapted over time using an online expectation-maximization (EM)algorithm.

U.S. Pat. No. 8,477,998, “Object tracking in video with visualconstraints,” describes tracking objects represented in a video bydetermining tracking states of the object based on a pose model, analignment confidence score, and an adaptive term value. The trackingstate defines a likely position of the object in the frame given theobject's likely position in a set of previous frames in the video.

Grassmanian Rank-One Update Subspace Estimation (GROUSE) is one methodthat can handle real-time subspace estimation from incomplete data, seeBalzano et al., “Online identification and tracking of subspaces fromhighly incomplete information,” 48th Annual Allerton Conference onCommunication, Control, and Computing (Allerton), pp. 704-711, Sep.2010. GROUSE uses rank-one updates of the subspace on a Grassmannianmanifold. However, GROUSE can get trapped at a local minima.

Parallel Subspace Estimation and Tracking by Recursive Least Squares(PETRELS) can also identify a low-dimensional subspace in real-time, seeChi et al., “Petrels: Parallel subspace estimation and tracking byrecursive least squares from partial observations,” IEEE Transactions onSignal Processing, vol. 61, no. 23, pp. 5947-5959, 2013. PETRELSminimizing, in parallel, a geometrically discounted sum of projectionresiduals on the data for each time step using a recursive procedurewith discounting for each row of the subspace matrix. Both GROUSE andPETRELS cannot correctly handle corrupted data and data subject tonon-Gaussian noise.

Grassmannian Robust Adaptive Subspace Tracking Algorithm (GRASTA) issimilar to GROUSE, see Cornell University, arXiv:1109.3827, Sep.September 2011. GRASTA also updates the Grassmannian manifold, butreplaces the l₂ cost function of GROUSE with a l₁-norm cost function.This cost function minimizes a sum of absolute errors, while correctlyhandling outliers in the data.

Another real-time method is Recursive Projected Compressive Sensing(ReProCS), see Qiu et al. “ReProCS: A missing link between recursiverobust PCA and recursive sparse recovery in large but correlated noise,”CoRR, vol. abs/1106.3286, 2011. ReProCS recursively projects data onto aorthogonal complement of the subspace followed by sparse recovery todetermine outliers. However, that method requires an accurate initialestimate of the subspace.

SUMMARY OF THE INVENTION

The embodiments of the invention provide a method and system forsegmenting and tracking time-varying content of videos in real-time. Thecontent can include a dominant component and a sparse components. Thedominant component can include, e.g., a largely stationary backgroundrepresented as a low-dimensional subspace. The sparse component caninclude moving objects represented by sparse vectors.

The video can be acquired from incomplete measurements and in thepresence of noisy outliers. The method minimizes a l₁-norm cost functionbetween the measurements and a projection of the measurements onto anestimate of the subspace.

Coefficients of the projection, and the sparse outliers are firstdetermined for the current estimate of the subspace using, for example,an alternating direction method of multipliers (ADMM), and the subspaceestimate is updated using a proximal point iterative procedure with anadaptive step-size parameter. The proximal point iterative procedure,similar to Newton's method, solves unconstrained smooth minimizationproblems involving high-dimensional data.

In one embodiment of the invention, the measurements are a sequence ofimages in a video. A set of one or more images are processed for eachtime step. For example, the set can be a group of pictures (GoP). Themethod characterizes the dominant stationary background of the video asthe low-dimensional subspace and separates the relatively largebackground from sparse objects that typically characterize smallermoving objects in the video, usually in the foreground.

In another embodiment of the invention, the measurements are motionvectors extracted from a sequence of images in a compressed video. Themotion vectors can represent the optical flow in the video. The methodaccording to this embodiment identifies and tracks a dominant opticalflow in the video using the low-dimensional subspace and separates thedominant optical flow from alternate optical flows, for example opticalflows that are different than the dominant optical flow.

In another embodient, the measured data corresponds to interest pointswith feature descriptors that are extracted and tracked in a streamingvideo. In one example, only a subset of the extracted interest pointsare tracked. An adjacency matrix is constructed to characteriseaffinities between the feature descriptors within an image and acrossreceived video images. The method according to this embodimentidentifies and tracks clusters of interest points that are representedby one or a union of subspaces that occupy a portion of the spectrum ofthe graph.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a method and system for processing a videoaccording to the embodiments of the invention; and

FIG. 2 is a block diagram of pseudocode of the method of FIG. 1.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The embodiments of our invention provide a method and system forsegmenting and tracking objects in a data stream that lie in lowdimensional subspaces using measurements of the data stream. Forexample, consider a sequence of images in a video stream where a largecollection of objects in the video have a dominant motion trajectorythat is static or changing slowly and where other objects in the videohave a different motion trajectory than the dominant trajectory.

In one embodiment, the method described in this invention separates astationary and relatively large background component of measurements, inthe form of images of a video arriving at a processor after the imagesare acquired, from typically sparse objects that characterize smallermoving objects in the video, usually in the foreground.

In another embodiment of the invention, the measurements are motionvectors extracted from a sequence of images in a compressed videopossibly corrupted with non-Gaussian noise. The motion vectors representthe optical flow in the video that tracks the motion of a largecollection of objects in the video. The method segments and tracks adominant optical flow using the low-dimensional subspace or a union oflow-dimensional subspaces.

In yet another embodiment, the measurements have missing data points.For example, if the motion vectors are extracted from only a subset of asequence of images in a compressed video. The method of this inventiondetermines the missing data points that correspond to the dominantoptical flow after identifying the low-dimensional subspace or a unionof low-dimensional subspaces.

As shown in FIG. 1, the method and system for processing measured data101, e.g., a sequence of images in a video 101 that is acquired of ascene 102 by a camera 103. The camera can be stationary or moving. Theimages can be processed in real-time, for example, at the same frame (24or 60 fps) as the images were acquired. The method operates in aprocessor 100 connected to memory and input/output interfaces by busesas known in the art. The memory is used to store the measured datarepresenting the images, as well as other data structures, such asvectors, matrices and parameters used by the method.

For first measured data 104 in the stream, e.g., a first image in thevideo sequence, an intial subspace matrix 131 and an initial step-sizeparameter 126 are initialized 105. A sparse component 111, in the formof a sparse vector, and subspace coefficients 112 are determined 110from the first measured data using an iterative solver, for example analternating direction method of multipliers (ADMM). Next, a change inthe subspace 121 is determined 120 according to the first measured data104, the sparse component 111, and the subspace coefficients 112. Anadaptive step-size parameter 126 is updated 125 according to the changein subspace 121 is updated 120. The subspace matrix 131 is then updated130 using the change in subspace 121 and the updated adaptive step-sizeparameter 126. For the second measured data and every subsequentmeasured data in the stream, the updated subspace matrix 131 and theupdated adaptive step-size parameter are used as the initial subspacematrix and the initial step-size parameter. The process is repeatediteratively until all the measured data 101 from the data stream areprocessed.

After the arrival of every new measured data from the stream, the movingobjects 108 as represented by the sparse vector have been separated fromthe background as represented in the current subspace matrix 121.

FIG. 2 shows pseudocode for the detailed steps of the method. Thevariables used by the pseudocode are defined below.

In one embodiment, the measured data correspond to features of interestpoints in a video sequence. A graph is constructed from the measureddata using feature descriptors, such as a Scale Invariant FeatureTransform (SIFT), a Fast Retina Keypoint (FREAK), a Binary RobustInvariant Scalable Keypoints (BRISK), etc., corresponding to theinterest points in order to assign edges and weights between theinterest points. The method then identifies one or a union of alow-dimensional subspaces that occupy a portion of a spectrum of thegraph and that characterises the dominant association between theinterest points. The method also segments the dominant association fromsparse or non-Gaussian distributed associations that exist in the graphspectrum.

Real-time Subspace Estimation

We descibe real-time estimation of the low-dimensional subspace matrix131 from incomplete streaming measurements 101, e.g., a compressedvideo, that may be corrupted with non-Gaussian noise. First, we describeour problem and define the notation used. Then, we describe minimizing al-₁-norm cost function between the measurements and their projectiononto the subspace to determine the subspace coefficients 112, and sparseoutliers 111. Then, the subspace is updated 130 using a proximal pointiterative procedure, based on using least squares estimation, whileupdating 125 the adaptive step-size parameter 126.

As advantages, our method does not restrict the subspace update to aGrassmannian manifold as in the prior art, and uses an adaptive stepsize. In addition, the method does not require an accurate initialestimate of the subspace, e.g., the subspace is set to a randomsubspace.

Augmented Lagrangian-Based Proximal Point Iterative Procedure

The method minimizes an augmented Lagrangian with the l₁-norm costfunction, and uses a smoothing term that maintains a proximity of theupdate to the previous subspace estimate over the variables (U_(t),s_(t), a_(t), y_(t)). An objective cost can be represented by

$\begin{matrix}{{{\mathcal{L}^{\prime}\left( {U_{t},s_{t},a_{t},e_{t},y_{t}} \right)} = {{s_{t}}_{1} + {y_{t}^{T}\left( {b_{t} - \left( {{U_{t}a_{t}} + s_{t} + e_{t}} \right)} \right)} + {\frac{\mu}{2}{{b_{t} - \left( {{U_{t}a_{t}} + s_{t} + e_{t}} \right)}}_{2}^{2}} + {\frac{\mu}{2}{{U_{t} - U_{t - 1}}}_{2}^{2}}}},} & (2)\end{matrix}$

where e_(t) is supported on the complement of the selection operatorΩ_(t), denoted Ω_(t) ^(c), such that Ω_(t)(e_(t))=0 and Ω_(t)^(c)(e_(t))=−Ω_(t) ^(c)(U_(t)a_(t)).

Equation (2) is non convex in the variables U_(t) and a_(t). Therefore,we follow the PETRELS and GRASTA approach of alternating theminimization over the variables (s_(t), a_(t), y_(t)) and then thevariables U_(t). By fixing U_(t), the minimizers of equation (2) areequal, i.e.,

$\begin{matrix}{\left( {s_{t},a_{t},y_{t}} \right) = {{\arg \; {\min\limits_{s,a,y}{\mathcal{L}\left( {s,a,y} \right)}}} = {\arg \; {\min\limits_{s,a,e,y}{{\mathcal{L}^{\prime}\left( {U_{t - 1},s,a,e,y} \right)}.}}}}} & (3)\end{matrix}$

Then, the subspace U_(t) is updated by taking a gradient step tominimize the function

$\begin{matrix}{{\mathcal{F}\left( U_{t} \right)} = {{\frac{1}{2}{{b_{t} - \left( {{U_{t}a_{t}} + s_{t} + e_{t}} \right)}}_{2}^{2}} + {\frac{1}{2}{{U_{t} - U_{t - 1}}}_{2}^{2}}}} & (4)\end{matrix}$

using the adaptive step-size μ.

Method

As shown by the pseudocode in FIG. 2, after prelimenaries, the firststage (steps 4-11) uses an iterative method, for example the ADMM, tosolve equation (3). The variables a_(t), s_(t), and y_(t) are determined(steps 7-10), until a termination condition is satisfied, by iteratingthe following sequence of updates:

a t k = U t - 1 †  ( b t - s t k - 1 - e t k - 1 + 1 μ t - 1  y t k -1 ) ,  e t k = - Ω t c  ( U t - 1  a t k ) ,  s t k = 1 μ t - 1  (b t - U t - 1  a t k - e t k - 1 μ t - 1  y t k - 1 ) ,  and   y tk = y t k - 1 + μ t - 1  ( b t - U t + 1  a t k - s t k - e t k ) ) ,( 5 )

where

_(τ)(x)=sign(x)·max{|x|−τ,0} denotes an element-wise soft thresholdingoperator with threshold τ, k indicates the iteration number, and †represents a Moore-Penrose pseudo-inverse of a matrix.

In the second stage (steps 12-17), the subspace U_(t) is updated (step19) by minimizing equation (4) using

$\begin{matrix}{{U_{t} = {\frac{\mu_{t - 1}}{\mu_{t}}\left( {U_{t - 1} + {\left( {b_{t} - s_{t} - e_{t}} \right)a_{t}^{T}}} \right)\left( {I_{r} + {a_{t}a_{t}^{T}}} \right)^{- 1}}},} & (6)\end{matrix}$

where I_(r) is an r×r identity matrix, and the step size μ_(t) 126 isupdated adaptively.

Adaptive Step-Size Parameter

For the adaptive step-size parameter, the method uses a regularizerμ_(t) to control the speed of convergence of the estimation of thesubspace 131. In particular, a smaller value of μ allows for fasteradaptability of U_(t) to a changing subspace, i.e., with a largerdescent direction, whereas a larger value of μ only allows a slowerchange in U_(t).

Consider a descent direction

D _(t)=(U _(t-1)+(b _(t) −s _(t) −e _(t))a _(t) ^(T))(I _(r) +a _(t) a_(t) ^(T))⁻¹ −U _(t-1),   (7)

and determine its projection onto an orthogonal complement of theprevious subspace estimate to obtain the change in subspace 121

G _(t)=(I−U _(t-1) U _(t-1) ^(†))D_(t)  (8)

Then, the adaptive step-size parameter μ_(t) 126 can be updated 125according to

${\mu_{t} = \frac{c\; 2^{- l}}{1 + \eta_{t}}},$

(9)where

${\eta_{t} = {\eta_{t - 1} + {{sigmoid}\left( \frac{\langle{G_{t - 1},G_{t}}\rangle}{{G_{t - 1}}_{F}{G_{t}}_{F}} \right)}}},$

and l ∈ {−1,0,1,2} are set according to predetermined thresholds forη_(t). Here, sigmoid(x)=f+2f/(1+e^(10x)) for some predefined f.

Similar to GRASTA, the intuition behind selecting such an update rulecomes from the idea that if two consecutive subspace updates G_(t-1) andG_(t) have the same direction, i.e., (G_(t-1), G_(t))>0, then the targetsubspace is still far from the current subspace estimate. Consequently,the updated step size μ_(t) should be smaller to allow for fastadaptability, which is achieved by increasing η_(t). Similarly, when(G_(t-1), G_(t))<0, the subspace update can oscilate around the targetsubspace and hence a larger μ_(t) is needed. Note that when the productof the norm of the subspace updates (∥G_(t-1)∥_(F)·∥G_(t)∥_(F)) is toosmall, e.g., smaller than 10⁻⁶, we assume that the current subspaceestimate is close to the target subspace, and we force η_(t) to decreaseby the magnitude of the sigmoid.

Although the invention has been described by way of examples ofpreferred embodiments, it is to be understood that various otheradaptations and modifications may be made within the spirit and scope ofthe invention. Therefore, it is the object of the appended claims tocover all such variations and modifications as come within the truespirit and scope of the invention.

We claim:
 1. A method for segmenting and tracking content in a videostream, wherein the video includes a sequence of images, comprising, foreach set of one or more images in the video stream, at each time step:determining measured data from the set; initializing a parameter and alow-dimensional subspace characterizing motion of the content in themeasured data, wherein the parameter is an adaptive step-size parameter;determining a sparse vector representing a sparse component thatcharacterizes the motion of the content different from the motion of thecontent characterized by the low-dimensional subspace; determining achange in the low-dimensional subspace for the measured data using aproximal point iteration and the parameter; updating the parameteraccording to the change; updating a low-rank subspace matrixrepresenting the low-dimensional subspace according to the change andthe parameter; and outputting the low-rank matrix representing thelow-dimensional subspace, and the sparse vector, wherein the steps areperformed in a processor connected to a memory storing the measureddata, and the video stream is acquired of a scene by a camera.
 2. Themethod of claim 1, wherein an estimate of the low-rank subspace matrixand an estimate of the parameter are available, and further comprising:determining a data misfit function to quantify a difference between themeasured data and a projection of the measured data onto the estimate ofthe low-rank matrix subspace; and determining the sparse vector and asubspace coefficient vector that minimize the data misfit function. 3.The method of claim 2, wherein the measured data are previouslyprocessed and new measured data are available, further comprising:determining the estimate of the low-rank subspace matrix and theestimate of the parameter from the updated low-rank subspace matrix andthe parameter of the previously processed measured data.
 4. The methodof claim 1, further comprising: determining an error component thatdetermines missing data points from the measured data using an estimateof the subspace.
 5. The method of claim 1, further comprising:determining the change by projecting a difference between the updatedlow-rank subspace matrix and an estimated value for previous measureddata onto an orthogonal complement of the low-dimensional subspace; andupdating the parameter by decreasing the parameter when thelow-dimensional subspace changes faster and increasing the parameterwhen the low-dimensional subspace changes slower.
 6. The method of claim1, wherein the measured data are selected from a group consisting ofpixel values of the set, optical flow motion vectors in the set, motiontrajectories of feature points in the set, and combinations thereof. 7.The method of claim 1, wherein the video stream is compressed, andfurther comprising: determining the measured data as compressed motionvectors of each image; determining the low dimensional subspace thatcharacterizes a dominant motion flow in the video stream; determiningthe sparse component that characterizes alternate motion motion flowsfrom the dominant motion flow; and outputting a low-rank matrixrepresenting the dominant motion flow, and the sparse vector thatrepresents the alternate motion flows.
 8. The method of claim 1, whereina graph is constructed from the measured data, further comprising:determining a graph matrix from one or a combination of a graphadjacency matrix, and a graph Laplacian; and determining thelow-dimensional subspace from one or a union of subspaces occupying aportion of a spectrum of the graph matrix.
 9. The method of claim 2,wherein the data misfit function is a l₁-norm cost function.